Goto

Collaborating Authors

 rebuttal period



During rebuttal period, we

Neural Information Processing Systems

We address your concern as follows. This clearly shows the advantage of our method. We answer your main questions as follows. Q1:"Do we need to commit ourselves to the OVR loss?...considering a loss function such as softmax cross entropy Y ou are absolutely correct! If convexity is not required (e.g., NN implementation), we can use more flexible multiclass loss and binary We will make this clear in the revision. Q2:"How to use the non-negative risk estimator in this problem?" We will add more elaborations about the formulation in the revision. Q3:"My question is have you tried different loss functions?" However, it does not converge in experiments. So we instead use sigmoid loss following Kiryo et al. [24]. Theorem 1 serves as a guide to choose binary loss for OVR scheme. Thus, a consistency guarantee (Theorem 1) is necessary. Thanks for the detailed review and helpful comments. We address your main concerns as follows. For the other minor issues, we will discuss in the paper and revise the paper according to your suggestions. We would like to revise the terminology in the revision if it is allowed. Q2:"Some of the claims made about prior work are not accurate.




Review for NeurIPS paper: Strongly Incremental Constituency Parsing with Graph Neural Networks

Neural Information Processing Systems

Weaknesses: The following are my concerns (questions) and confirmations of the proposed method. I understand that the number of actions required to parse a sentence for the proposed method is n, where the number of tokens in the sentence is n . However, the computational cost for one action seems relatively very expensive comparing with the existing transition-based algorithm, such as standard shift-reduce parser. Therefore, I suspect that the actual runtime of parsing a single sentence takes much larger than the conventional methods. Regardless of my suspicion is correct or not, the experimental results in the current version does not answer this point.


Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025

Thakkar, Nitya, Yuksekgonul, Mert, Silberg, Jake, Garg, Animesh, Peng, Nanyun, Sha, Fei, Yu, Rose, Vondrick, Carl, Zou, James

arXiv.org Artificial Intelligence

Peer review at AI conferences is stressed by rapidly rising submission volumes, leading to deteriorating review quality and increased author dissatisfaction. To address these issues, we developed Review Feedback Agent, a system leveraging multiple large language models (LLMs) to improve review clarity and actionability by providing automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers. Implemented at ICLR 2025 as a large randomized control study, our system provided optional feedback to more than 20,000 randomly selected reviews. To ensure high-quality feedback for reviewers at this scale, we also developed a suite of automated reliability tests powered by LLMs that acted as guardrails to ensure feedback quality, with feedback only being sent to reviewers if it passed all the tests. The results show that 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers. This suggests that many reviewers found the AI-generated feedback sufficiently helpful to merit updating their reviews. Incorporating AI feedback led to significantly longer reviews (an average increase of 80 words among those who updated after receiving feedback) and more informative reviews, as evaluated by blinded researchers. Moreover, reviewers who were selected to receive AI feedback were also more engaged during paper rebuttals, as seen in longer author-reviewer discussions. This work demonstrates that carefully designed LLM-generated review feedback can enhance peer review quality by making reviews more specific and actionable while increasing engagement between reviewers and authors. The Review Feedback Agent is publicly available at https://github.com/zou-group/review_feedback_agent.


Reviews: The Thermodynamic Variational Objective

Neural Information Processing Systems

The paper connects variational inference with thermodynamic integration, so that the data log-likelihood can be formulated as a 1D integration of the instantaneous ELBO in a unit interval. By applying a left Riemann sum, TVO, a novel lower bound for the marginal log likelihood, is derived in which the traditional variational ELBO is recovered when only one partition is used. The authors then design an importance-sampling-based gradient estimator to optimize the objective, and compare with other methods on both discrete and continuous deep generative models. Originality and Significance: the formulation of TVO is an interesting idea. Better optimization methods than the importance-sampling-based approach are worth further exploring.


Some Ethical Issues in the Review Process of Machine Learning Conferences

Russo, Alessio

arXiv.org Machine Learning

Recent successes in the Machine Learning community have led to a steep increase in the number of papers submitted to conferences. This increase made more prominent some of the issues that affect the current review process used by these conferences. The review process has several issues that may undermine the nature of scientific research, which is of being fully objective, apolitical, unbiased and free of misconduct (such as plagiarism, cheating, improper influence, and other improprieties). In this work, we study the problem of reviewers' recruitment, infringements of the double-blind process, fraudulent behaviors, biases in numerical ratings, and the appendix phenomenon (i.e., the fact that it is becoming more common to publish results in the appendix section of a paper). For each of these problems, we provide a short description and possible solutions. The goal of this work is to raise awareness in the Machine Learning community regarding these issues.